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ABSTRACT 

The  higher  a  military  person  goes  in  the  ranks,  the  sooner  the  transfer,  meaning  the 
eommanders  who  make  the  highest  level  deeisions  have  been  in  a  particular  assignment 
the  shortest  amount  of  time.  To  acquire  and  maintain  the  maximum  level  of  efficiency 
and  productivity  as  quickly  as  possible,  the  expertise  of  commanders  must  be 
understood  so  that  appropriate  decision  support  can  be  developed.  Cognitive  task 
analysis  methods,  including  knowledge  elicitation,  can  be  applied  to  uncover  the 
information  about  how  a  domain  practitioner  works.  Such  information  on  practitioner 
behavior  provides  valuable  insight  and  data  for  incorporation  into  models  used  for 
training  and  development  of  automated  support.  However,  knowledge  elicitation 
methods  face  a  number  of  challenges  such  as  grounding  in  context,  limited  accessibility 
to  experts  and  tasks,  being  labor  intensive  and  time  consuming,  and  difficulties  with 
repeatability.  In  this  study,  a  critiquing  methodology  was  investigated  in  its  ability  to 
address  these  challenges.  This  baseline  study  involved  a  novice  intelligence  analyst 
performing  a  basic  analysis  task.  Then,  six  experts  with  various  backgrounds  critiqued 
the  novice’s  process.  The  results  suggest  that  the  critiquing  method  addresses  the 
challenges  of  knowledge  elicitation  methods  and  can  be  applied  to  understanding 
command  and  control. 


INTRODUCTION 


One  given  fact  in  the  military  is  that  the  soldier,  the  sailor,  and  the  airman  will  be 
transferred.  The  higher  a  military  person  goes  in  the  ranks,  the  sooner  the  transfer, 
which  means  the  commanders  who  make  the  highest  level  decisions  have  been  in  a 
particular  organization  and  assignment  the  shortest  amount  of  time.  This  method  of 
continually  transferring  military  members  to  new  assignments  on  the  average  of  every 
three  years  positively  contributes  to  rounding  out  the  knowledge  and  experience  of  the 
military  person.  However,  each  particular  assignment,  especially  command  and 
control,  requires  specific  expertise  beyond  general  knowledge.  Even  less  frequently 
encountered  are  true  direct  military  conflict  assignments  that  have  an  inordinate  level  of 
dynamic,  stressful,  and  risky  challenges.  To  acquire  and  maintain  the  maximum  level  of 
efficiency  and  productivity  in  a  particular  assignment,  including  high-risk  assignments 
such  as  Joint  Forces  Air  Component  Commander  (JFACC),  as  quickly  as  possible,  the 


expertise  of  eommanders  must  be  understood  so  that  appropriate  deeision  support, 
ineluding  training,  ean  be  developed.  Cognitive  task  analysis  (CTA)  reveals  those 
speeifie  faeets  of  expertise  that  domain  praetitioners,  sueh  as  eommanders,  utilize 
ineluding  the  information  requirements  for  deeision  making.  Therefore,  eognitive  task 
analysis  needs  to  be  used  with  eommanders  to  understand  their  deeision  ehallenges  and 
problems  so  that  appropriate  deeision  support  ean  be  developed  to  aid  in  eurrent 
deeision  making  and  to  fortify  against  lost  deeision  rationale  and  understanding, 
espeeially  after  personnel  turnover. 

Cognitive  task  analysis  is  part  of  the  field  of  eognitive  engineering  that  looks  at 
knowledge,  strategies,  and  goal  struetures  that  underlie  observable  work  performanee. 
CTA  methodologies  are  used  to  diseover  the  basis  of  expertise  that  domain 
praetitioners  use  to  perform  tasks,  but  are  unable  to  reliably  artieulate  when  asked 
direetly.  Speeifieally,  they  identify  ineffeetive  strategies  that  lead  to  poor  performanee 
(i.e.,  a  model  of  mistakes  that  “noviees”  make),  as  well  as  adaptive  strategies  that  have 
been  developed  by  highly  skilled  practitioners  to  cope  with  task  demands  (i.e.,  a  model 
of  “expert”  performance).  A  variety  of  these  approaches  have  been  developed  for 
gaining  insight  to  the  performance  of  highly  skilled  practitioners.  They  range  from 
field  studies  where  practitioners  are  observed  in  actual  work  settings  (Hutchins,  1995) 
to  critical  incident  analyses  (Klein,  Calderwood,  and  Clinton-Cirocco,  1986)  to 
analyses  of  performance  under  simulated  conditions  (Sarter  and  Woods,  2000)  to 
observation  of  practitioner  performance  under  highly  controlled  conditions  (Lesgold  et 
al,  1988).  Each  of  these  methods  has  advantages  and  disadvantages,  and  can  be 
combined  to  obtain  converging  evidence  in  order  to  “bootstrap”  an  understanding  of  the 
sources  of  expertise  in  a  domain  (Potter  et  al,  2000). 

A  traditional  beginning  to  cognitive  task  analysis  is  to  gain  some  understanding  about  a 
domain  through  research,  reading  documentation,  and  other  indirect  means.  Then  direct 
knowledge  acquisition  methods  are  used  to  uncover  information  about  how  a 
practitioner  works  in  a  field  of  practice,  including  the  work  done  'in  the  head,'  from  the 
actual  practitioner.  These  direct  methods,  known  as  knowledge  elicitation  methods, 
include  a  variety  of  interview  techniques  and  different  ways  to  observe  task 
performance.  Although  undoubtedly  any  of  the  knowledge  elicitation  methods  would 
reveal  some  knowledge  about  a  domain,  there  are  considerations  on  choosing  the 
technique  most  likely  to  be  effective  for  a  particular  domain.  Considerations  include 
whether  the  technique  can  authentically  capture  information  about  the  domain,  whether 
the  results  of  the  technique  are  commensurate  with  the  expended  resources,  whether  the 
resulting  products  are  sufficiently  documented,  and  whether  the  method  generates  ideas 
for  domain  support  (Potter  et  al,  2000).  Good  techniques  encourage  shared 
communication  between  the  investigator  and  the  practitioner,  facilitate  an 
unconstrained  expression  of  knowledge,  provide  good  data  quality,  and  are  able  to  be 
replicated  (McNeese  et  al,  1995;  Hoffman,  Crandall  and  Shadbolt,  1998). 

Eliciting  knowledge  from  experts  is  not  easy.  Eor  one,  careful  thought  must  be  given  to 
planning  and  developing  the  details  of  the  method  to  be  applied  to  ensure  the  elicited 
knowledge  is  grounded  in  context.  Another  issue  is  potential  difficulties  gaining  access 


to  both  experts  and  to  meaningful  tasks.  Aeeess  to  experts  ean  be  an  issue  if  there  are 
only  a  few  experts  performing  the  task  of  interest,  if  there  are  physieal  aeeess 
ehallenges,  sueh  as  being  in  a  restrieted  area,  or  if  the  experts  are  reluetant  to  be 
evaluated  beeause  of  possible  repereussions.  Aeeess  to  meaningful  tasks  ean  be  an 
issue  if  the  meaningful  events  oeeur  at  a  low  frequeney,  there  is  unpredietability  of 
target  events,  or  there  are  physieal  aeeess  limitations.  For  a  third,  the  proeess  is 
laborious  and  time  eonsuming.  Hours  are  required  not  only  for  developing  the  details 
of  the  session  for  elieiting  knowledge  but  also  to  proeess  the  extraeted  information  into 
a  useable  form.  A  last  issue  is  repeatability,  whieh  supports  making  sense  of  the 
gathered  information  by  allowing  eomparisons  between  sessions. 

Command  and  eontrol  (C2)  presents  a  unique  domain  for  elieiting  knowledge.  For  one, 
there  are  only  so  many  C2  positions  and  while  there  are  some  similarities  in  the  tasks 
performed,  their  differenees  are  greater  than  their  similarities.  This  situation 
eontributes  to  both  interesting  task  and  expert  aeeess  issues.  Then,  dynamie,  eritieal  C2 
positions  sueh  as  the  JFACC  only  exist  during  eonfliet  times  when  the  eommander  is 
extremely  oeeupied  with  the  job  at  hand.  Opportunity  to  eapture  expertise  on  partieular 
eognitively  demanding  C2  tasks  under  these  eonditions  is  therefore  extremely  limited. 
Finally,  repeatability  and  grounding  in  eontext  are  ehallenges  as  the  eontext  of  eonfliots 
has  many  variables  that  must  be  eonsidered  by  the  eommander,  ineluding  eultural  and 
behavioral  aspeets  of  the  eountry  in  foeus. 

Interviews  and  performanee  observations  are  knowledge  elieitation  methods  that  ean 
address  the  ehallenges  in  different  ways  and  to  different  degrees. 

•  Grounding  in  Context: 

Interviews  provide  an  opportunity  for  a  domain  praetitioner  to  retrospeetively  tell  how 
he  performs  work.  This  means  that  the  praetitioner  is  not  physieally  grounded  in 
eontext  and  that  a  definite  ehallenge  exists  to  have  the  praetitioner  be  mentally 
grounded.  The  CTA  investigator  needs  to  have  deep  understanding  of  the  work,  whieh 
requires  extensive  pre-session  work,  so  that  the  praetitioner  is  provided  realistie, 
stimulating  questions  to  reveal  his  knowledge.  The  interviews  foeus  on  telling  stories 
that  ean  bring  out  interesting  knowledge  that  the  praetitioner  might  not  even  realize  is 
interesting.  As  the  questions  guide  the  session,  the  revealed  knowledge  is  based  on 
what  the  investigator  has  predetermined  will  be  interesting.  Performanee  observations 
are  prospeetive  as  the  praetitioner  is  revealing  what  he  is  about  to  do  as  he  shows  what 
he  does.  In  observations,  then,  the  praetitioner  is  both  physieally  and  mentally  grounded 
in  eontext  as  he  is  doing  realistie  work.  Being  bound  in  a  realistie  eontext  allows  for 
natural  triggering  of  domain  knowledge  and  promoting  aeeurate  verbalizations. 
Observations  require  that  the  CTA  investigator  has  prior  knowledge  of  the  work  to  be 
prepared  to  eapture  interesting  events.  But  as  the  session  is  not  guided,  the  investigator 
does  not  need  extensive  prior  knowledge  on  what  will  be  interesting,  whieh  ean  reduee 
the  pre-session  planning.  Another  advantage  of  having  a  less  guided  session  is  that 
unexpeeted  opportunities  may  naturally  arise  that  the  investigator  ean  eapture. 

•  Aeeess  Issues: 

Physieal  aeeess  issues  ean  be  addressed  with  interviews,  as  the  CTA  investigator  does 
not  neeessarily  have  to  be  in  the  praetitioner's  work  area  for  the  interview  session. 


Performance  observation  using  simulations  can  also  overcome  physical  access  issues. 
Meaningful  task  access  issues  are  addressed  with  well-planned  interviews  by  directly 
asking  about  such  events.  Performance  observations  do  not  ensure  access  to  meaningful 
events.  If  an  unexpected,  interesting  event  does  occur,  though,  the  investigator  is 
available  to  gather  data.  If  the  performance  observation  is  a  simulation,  the  simulation 
can  be  based  on  intentional  interesting  events,  such  as  a  crisis. 

•  Laborious  and  Time  Consuming: 

With  the  interview  technique,  the  questions  provide  a  framework  for  the  CTA 
investigator  to  know  ahead  of  time  when  interesting  information  will  be  provided 
which  can  decrease  the  time  to  process  the  extracted  information.  While  performance 
observations  might  require  less  time  in  the  planning  phase,  the  time  to  process  the 
extracted  information  is  increased,  as  the  investigator  has  not  predetermined  what  will 
be  interesting. 

•  Repeatability: 

Interviews  allow  repeatability.  Observation  of  simulations  supports  repeatability  but 
repeatability  of  actual  work  tasks  is  dependent  upon  the  domain. 

In  summary,  interview  elicitations  and  performance  observations  have  different 
strengths  and  weaknesses.  The  strengths  of  interviews  are  they  overcome  some  of  the 
expert  and  task  access  issues,  they  are  less  laborious  and  time  consuming  in  processing 
the  gathered  data,  and  they  allow  repeatability.  The  weaknesses  of  interviews  are  they 
are  not  fully  grounded  in  context,  the  practitioner  is  retrospectively  telling  how  he  does 
a  task,  they  require  more  pre-session  understanding  to  ensure  interesting  and  relevant 
issues  are  discussed,  and  they  do  not  provide  for  flexibility  in  gathering  data  on 
unexpected  events.  The  strengths  of  performance  observations  are  they  are  grounded  in 
physical  and  mental  context,  the  practitioner  is  prospectively  showing  and  discussing 
how  he  does  a  task,  they  require  less  pre-session  planning,  and  they  allow  the 
investigator  to  take  advantage  of  unexpected  events.  The  weaknesses  of  performance 
observations  are  they  are  laborious  and  time  consuming  to  process  the  gathered  data, 
they  may  have  physical  and  task  access  issues,  and  they  do  not  necessarily  allow 
repeatability. 

Interviews  and  observation  methods  have  been  useful  in  revealing  how  practitioners 
work  in  various  domains  and  can  be  used  in  C2.  However,  developing  another 
methodology  that  combines  the  strengths  and  addresses  the  weaknesses  of  the  two 
methods  would  be  beneficial  to  the  field  of  CTA  and  can  help  address  the  particular 
challenges  of  capturing  C2  expertise.  This  paper  discusses  a  methodology  that 
proposes  to  address  that  goal,  critiquing  by  an  expert.  'Expert'  is  used  as  shorthand  for 
an  experienced,  knowledgeable  practitioner,  which  is  a  more  accurate  categorization  of 
the  practitioners  who  participated  in  this  study.  A  true  expert  is  a  practitioner  who  is 
distinguished  or  brilliant  in  the  field,  highly  regarded  by  peers,  whose  judgments  are 
uncommonly  accurate  and  reliable,  whose  performance  shows  consummate  skill  and 
economy  of  effort  and  who  can  deal  effectively  with  rare  or  'tough'  cases  (Hoffman,  et 
al,  1995).  'Novice'  is  similarly  used  as  shorthand  for  a  person  who  is  new  to  a  domain 
although  he  may  have  skills  that  translate  from  prior  experience  in  other  domains. 


Figure  1  illustrates  some  of  the  ways  that  interviewing,  observation,  and  eritiquing 
elieitation  methods  differ.  With  interviewing,  the  expert  tells  information  to  the 
investigator,  often  using  stories  based  on  past  oases  and  experienoes.  While  being 
questioned  by  the  investigator,  the  expert  as  storyteller  might  reveal  not  only  how  he 
handled  a  partioular  ease,  but  also  have  further  oomments  on  how  that  ease  ohanged  his 
later  work  praotioes.  The  value  of  the  results  from  interviews  is  mainly  a  funotion  of 
the  probing  skill  of  the  investigator  and  how  well  the  interviewee  understands  what 
type  of  data  is  sought.  With  direot  observation,  the  investigator  watohes  the  expert  as  a 
domain  praotitioner  performing  aotual  or  simulated  tasks.  The  value  of  the  results  is 
mainly  a  funetion  of  how  realistie  the  seenarios  and  performanees  are  and  how  well  the 
observed  events  stress  the  eognitive  system  in  order  to  reveal  leverage  points  for 
improvement.  With  eritiquing,  the  expert  evaluates  the  performanee  of  another 
praetitioner.  This  sets  up  a  situation  where  the  domain  expert  is  not  eoneerned  about 
his  performanee  evaluation.  In  addition,  the  teehnique  relies  less  on  the  investigator’s 
probing  skills  and  domain  knowledge.  The  value  of  the  results  is  mainly  a  funetion  of 
the  ability  of  the  investigator  to  analyze  the  diverse  data  that  is  eolleeted,  whieh  is  of 
varying  reliability  and  validity  and  not  always  struetured  by  a  story  or  seenario. 
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Figure  1.  Comparison  of  Knowledge  Elieitation  Methods 

Critiquing  of  a  praotitioner  by  an  expert  somewhat  relates  to  the  researoh  on  expert 
eritiquing  systems  but  is  at  the  same  time  very  different.  Expert  eritiquing  systems 


(Fischer,  et  al,  1991),  or  critics,  are  a  elass  of  expert  systems  that  receive  as  input  the 
statement  of  the  problem  and  the  user-proposed  solution.  Likewise,  critiquing  as  a  CTA 
methodology  is  done  by  having  an  expert  practitioner  be  presented  with  the  task  of 
interest  and  the  noviee's  strategy.  Automated  erities  recognize  possible  errors,  form 
persuasive  critieisms  against  what  might  be  a  chosen  sub-optimal  proposed  solution, 
and  adapt  to  the  situation  and  any  lessons  learned.  Critics  are  silent  if  no  error  is 
identified.  Critiquers  for  the  CTA  methodology  are  encouraged  to  verbalize  on  possible 
errors  as  well  as  good  actions  taken  by  the  novice  and  to  make  eomments  about  what 
they  would  have  done  had  they  been  working  the  task. 

METHODOLOGY 

As  a  baseline  study  into  eritiquing,  this  study  was  done  in  the  military  intelligenee 
analysis  domain  involving  a  newly  assigned  officer  performing  a  basic  domain  task. 
Then,  the  task  and  a  presentation  of  the  newly  assigned  officer’s  performance  on  the 
task  were  presented  to  six  experienced  domain  praetitioners  for  critiquing.  The  domain 
praetitioners’  task  was  to  critique  the  newly  assigned  offieer’s  performance,  a  task  that 
is  similar  to  the  normal  mentoring  done  by  seasoned  officers.  The  newly  assigned 
offieer’s  performance  was  used  as  a  cue  to  the  experts  for  discussion  on  what  the 
experts  consider  correct  and  incorrect  actions  and  to  comment  on  what  they  would  have 
done  had  they  been  working  the  domain  task  based  on  past  experiences.  A  second 
novice  was  used  for  method  verification. 

The  interest  for  this  study  was  two-fold:  to  explore  eritiquing  as  a  knowledge 
elicitation  methodology  and  to  investigate  the  domain  of  intelligence  analysis  with  a 
focus  on  revealing  insights  that  might  be  used  for  developing  support  models.  The 
breadth  of  intelligence  analysis  work  ineludes  analyzing  various  types  of  media,  such  as 
physical  artifacts  and  photography  as  well  as  text,  and  reporting  in  various  ways  from 
short  briefings  to  long  reports.  To  address  both  the  methodology  and  domain  interests 
this  study  was  limited  to  investigating  the  intelligenee  analysts’  basic  process  on  textual 
data  of  information  selection,  corroboration/resolution,  and  story  construction.  The 
specific  task  statement  was:  “In  1996,  the  European  Space  Agency  lost  a  satellite 
during  the  first  qualification  launch  of  new  roeket  design 
Give  a  short  briefing  about  the  basic  facts  of  the  ineident:  when  it  was,  why  it 
occurred,  and  what  the  immediate  impaets  were.”  This  task  was  ehosen  after 
eonsultation  with  domain  experts  because  the  task  was  very  representative  of  actual 
work  accomplished  in  military  intelligence  analysis.  The  available  electronic  data  set 
contained  about  2000  text  doeuments  and  had  enough  information  to  provide  a  detailed 
answer  to  the  written  task.  Approximately  60%  were  ‘on  target’  in  that  they  contained 
information  that  could  directly  add  to  answering  the  task.  Approximately  35%  gave 
context  but  did  not  eontain  information  speeific  to  the  task.  The  last  5%  were  irrelevant 
to  the  task. 

First,  a  novice  was  given  a  task  to  perform  and  asked  to  talk  aloud  during  the  task.  The 
task  was  to  answer  speeifie  questions  by  using  information  in  the  provided  eleetronic 
data  set.  The  novice’s  proeess  of  developing  a  response  was  captured  by  video  and 


audio  taping  with  his  consent.  To  present  the  noviee’s  proeess  to  the  experts,  the 
eomments  made  by  the  noviee  during  the  analysis  were  transeribed  into  a  seript.  In 
addition,  the  notes  whieh  the  noviee  made  during  the  analysis  were  reereated  into 
sueeessive  PowerPoint  slides  to  show  the  development  of  the  note  artifaet  over  time. 
Finally,  sereen  shots  were  made  of  his  query  responses  and  hard  eopies  were  made  of 
the  doeuments  that  he  reviewed.  The  slides,  sereenshots  and  doeuments  were  shown  at 
eoordinated  times  with  the  seript.  A  seeond  noviee  also  performed  the  task  and  the 
same  data  was  eolleeted  to  verify  results.  Both  noviees  were  reeently  eommissioned 
U.S.  Air  Foree  seeond  lieutenants.  The  first  had  reeeived  a  B.S.  in  physies  and  the 
seeond  noviee  had  a  B.S.  in  aerospaee  engineering.  Neither  had  done  intelligenee 
analysis  and  the  only  task  that  either  had  performed  whieh  was  similar  to  the  study  task 
was  researeh  in  the  aeademie  environment. 

Six  experieneed  analysts  (referred  to  as  E5  to  ElO  beeause  there  were  four  pilot  study 
partieipants)  partieipated  in  this  study  and  had  been  nominated  by  their  organization  to 
partieipate  as  representative  experts.  As  ean  be  seen  in  Table  1,  their  baekgrounds 
varied  with  respeet  to  years  as  an  analyst,  prior  knowledge  and  edueational  baekground. 
Prior  knowledge  refers  to  how  informed  the  experts  were  of  the  partieular  satellite 
launeh  ineident  prior  to  the  eritiquing  session.  Two  of  the  analysts  had  no  knowledge 
about  the  launeh  aeeident  or  the  partieular  task  prior  to  their  eritiquing  session.  Two  of 
the  analysts  did  not  have  knowledge  about  the  ineident  from  their  work  experienee  but 
were  given  the  task  statement  and  the  doeuments  that  the  first  noviee  used  in  answering 
the  task  statement  prior  to  their  eritiquing  session.  The  final  two  analysts  had  both 
knowledge  of  the  ineident  from  their  work  experienee  and  were  given  the  task 
statement  and  the  noviee’s  doeuments  prior  to  their  eritiquing  session. 

When  eaeh  expert  reported  for  his  session,  the  problem  task  was  read  to  the  expert. 
The  experts  were  asked  to  think  aloud  as  the  noviee’s  proeess  was  revealed.  Then,  the 
seript  of  the  first  noviee’s  proeess  was  read  with  pauses  to  eneourage  eommenting.  As 
the  seript  was  read,  the  PowerPoint  viewgraphs  of  the  note  artifaet,  sereen  shots  of 
query  responses  and  hard  eopies  of  the  retrieved  doeuments  were  shown.  The  experts’ 
eomments  about  the  noviee’s  proeess  were  eaptured  on  audio  and  videotape.  The  two 
experts  who  had  full  prior  knowledge  also  eritiqued  the  presentation  of  the  seeond 
noviee’s  proeess. 

The  data  to  be  analyzed  for  the  study  was  the  experts’  protoeols  from  eritiquing  the 
noviees.  This  data  was  iteratively  analyzed  using  a  proeess  traeing  methodology 
(Woods,  1993).  Eirst,  a  set  of  detailed  protoeols  from  eaeh  expert  partieipanfs  proeess 
for  the  primary  noviee  was  eonstrueted.  The  protoeols  were  broken,  by  partieipant,  into 
the  frameworks  of  information  seleetion,  eorroboration/resolution,  and  story  eonstruet. 
A  eomparison  of  the  results  of  the  primary  noviee  was  made  to  the  results  from  the 
seeond  noviee  to  substantiate  the  findings. 


Table  1.  Background  Information  on  Experienced  Analysts 


Participant 

Gender 

Years  as 
Analyst 

Prior  Knowledge 

University  Degrees 

E5 

M 

8 

None 

BSEE 

MSEE 

MS  in  Studies  of  Futures 

E6 

M 

21 

None 

BS  in  Aero 

E7 

M 

12.5 

Full 

BS  in  Aero, 

MA  in  Strategic  Intelligence 

E8 

M 

18 

Partial 

BA  in  Business  Mgt 

E9 

M 

32 

Partial 

BS  in  Physics 

BS  in  Math 

MS  in  Nuclear  Physics 

ElO 

M 

12 

Full 

BA  in  Psychology 

MA  in  Intem'l  Relations 

RESULTS 

Analysis  of  gathered  information  revealed  that  the  three  main  tasks  of  information 
selection,  corroboration/resolution  and  story  construction  could  be  broken  down  further 
into  other  subtasks.  Information  selection  includes  problem  definition,  applying  a 
search  strategy,  and  evaluating  query  results.  Corroboration  and  resolution  includes 
building  a  hypothesis  or  hypotheses  against  which  to  judge  information  and  evaluating 
information.  Story  construction  involves  deciding  on  the  best  hypothesis  and  putting 
the  story  together.  This  refined  breakdown  was  used  for  analysis  of  trends. 

Tables  2-5  show  the  results  of  E7  and  ElO  on  both  novices  as  representative  of  the  data 
gathered  on  all  six  experts.  Although  there  was  not  complete  overlap  in  the  comments 
by  the  experts  on  both  novices,  commonality  existed.  Expert  7  had  two  comments  for 
each  on  focusing  on  the  problem  (Table  2:  #1  and  2  and  Table  3:  #1  and  2)  and  one  for 
each  on  the  need  to  talk  to  others  (Table  2,  #6  and  Table  3:  #5),  of  needing  direct 
information  sources  (Table  2:  #9  and  Table  3:  #6),  the  necessity  of  multiple  sources 
(Table  2:  #5  and  Table  3:  #7)  and  the  need  to  know  your  audience  (Table  2:  #11  and 
Table  3:  #11).  Expert  10  had  two  comments  for  trainee  1  (Table  4:  #3  and  #6)  and  one 
for  trainee  2  (Table  4:  #3)  on  using  metadata  in  titles;  one  comment  each  for  trainee  1 
and  for  trainee  2  on  keeping  task  focus  (Table  4,  #1  and  Table  5:  #4),  looking  for 
independent  reviews  (Table  4;  #11  and  Table  5:  #9),  using  several  sources  (Table  4; 
#12  and  Table  5:  #8)  and  on  including  implications  in  the  assessment  (Table  4:  #19  and 
Tables:  #11). 


Table  2.  Critiquing  Comments  of  E7  on  Novice  1 


Analyst 

Information  Selection 

Corroborate/Resolution 

Story  Construction 

E7  on  #1 

1.  Understand 
what  the 
question  really 
is 

2.  Have  a  goal  in 
mind 

3.  Need  to  check 
intelligence 
sources  not 
just  open 
source 

4.  Be  aware  of  directed 
sources,  where  they  only 
put  in  what  they  want 
you  to  believe 

5.  Need  multiple  sources  to 
confirm  data 

6.  Talk  to  other  people  to 
get  their  take 

7.  Reports  six  months  or  so 
after  an  event 
(depending  on  the  event) 
probably  have  more 
accurate  information 
than  those  immediately 
around  event 

8.  Take  open  source  with  a 
grain  of  salt  -  might  be 
on  soap  box,  misled 
themselves,  intentionally 
misleading  audience 

9.  Human  sources  have  to 
have  direct  knowledge 
for  creditability 

10.  Get  familiar 
before 
formulating 

1 1 .  Need  to  know  the 
audience  is  very 
true 

12.  Senior  people 
want  pictures, 
graphs 

13.  Include  basics  if 
the  audience 
might  not  be  fully 
informed 

14.  Admit  what  you 
don't  know 

15.  You  have  to  be 
right  with  what 
you  say 

16.  Never  contradict 
yourself 

Table  3.  Critiquing  Comments  of  E7  on  Novice  2 


Analyst 

Information  Selection 

Corroborate/Resolution 

Story  Construction 

E7  on  #2 

1 .  Understand 
the  problem 

2.  Keep  full  task 
in  mind  as 
you  read 

3 .  Look  for 
patterns  in  the 
title  list 

4.  Don't  build 
results  on  one 
search. 

5.  Talk  to  other  analysts  to 
see  to  discuss  the 
problem 

6.  What  source  information 
comes  from  is  very 
important,  loses  validity 
if  2nd  or  3rd  hand 
information 

7.  It's  necessary  to 
corroborate  information, 
might  not  use  if  only  in 
one  source. 

8.  Carefully  construct 
story 

9.  If  you  mention 
something,  be 
prepared  to  fully 
explain 

10.  Senior  people  want 
few  words  and  lots 
of  pictures  and 
graphs  in  a 
summary  version 

1 1 .  Know  the  audience 

12.  Management  wants 
to  know  the 
implications 

Table  4.  Critiquing  Comments  of  ElO  on  Novice  1 


Analyst 

Information  Selection 

Corroborate/Resolution 

Story  Construction 

ElO  on 
#1 

1 .  Can  get  wrong 
focus  if  don't 
correctly  define 
problem 

2.  Go  back  to  the 
person  asking 
question  to 
resolve 
ambiguity 

3.  If  using  on¬ 
screen  views  of 
title  list,  make 
sure  window  is 
large  enough  to 
notice  patterns 
of  large  part  of 
data  set 

4.  Read  titles  to 
get  a  feel  for  the 
data  set 

5.  Value  of  data 
dependent  upon 
source 

6.  Use  meta  data 
in  title  list,  that 
is,  for  example, 
the  fact  one 
name  shows  up 
a  multitude  of 
times 

7.  Ask  other  analysts  for 
opinions 

8.  Will  have  to  evaluate 
data,  not  just  read  it 

9.  Language  can  be 
considered  a  high  level 
determinant  during  data 
evaluation 

10.  Watch  for  biased 
reporting 

11.  Be  aware  of 
information  that  is 
actually  only  repeating 
another  source 

12.  Don't  base  assessment 
on  only  a  few 
documents 

13.  Be  aware  complete 
details  might 
intentionally  not  be 
revealed 

14.  Be  aware  information 
might  not  be  directly 
stated 

15.  Look  for  patterns 

16.  Sometimes  setting 
down  a  timeline 
helps 

17.  Use  external 
memory,  if 
necessary,  to  jog 
memory 

18.  Diagram  the  info 
if  possible 

19.  Include 
implications  in 
assessment 

20.  Have  list  of 
sources  to  support 
assessment 

2 1 .  Accept  that  time 
and  resource 
constraints  affect 
the  possible  result 

Table  5.  Critiquing  Comments  of  ElO  on  Novice  2 


Analyst 

Information  Selection 

Corroborate/Resolution 

Story  Construction 

TO  on  #2 

1 .  Do  more  than 
one  search 

2.  Look  for 
implications 

3.  Use 
information 
in  titles  but 
don't  fully 
judge 
documents 
with  that 

4.  Keep  focused 
on  task 

5.  Look  for 
information 
over  time 

6.  Compare  information 
over  time  to  look  for 
changes. 

7.  Always  have  to  evaluate 
sources. 

8.  Use  several  sources 

9.  Look  for  independent 
reviews 

10.  Be  prepared  for 
in-depth  questions 

11.  Include 
implications. 

DISCUSSION 


Reviewing  the  protoeols  indicated  that  critiquing  did  address  the  challenges  of  proper 
grounding  in  context,  limited  accessibility  to  experts  and  tasks,  being  labor  intensive 
and  time  consuming,  and  difficulties  with  repeatability.  Grounding  in  context  is 
important  to  encourage  comments  that  are  true  and  interesting  for  the  domain  being 
investigated.  The  task  and  the  domain  used  for  this  initial  investigation  into  the 
critiquing  methodology  allowed  uncomplicated  means  for  grounding  in  context.  Task 
accessibility  issues  are  addressed  by  the  critiquing  method  used  in  this  study  as  once 
the  novice’s  process  was  captured,  the  re-created  novice  performance  could  and  was 
used  multiple  times.  Physical  access  to  experts  was  overcome  because  the  critiquing 
session  was  held  outside  of  normal  work  areas  while  still  accessing  the  tool  set  used 
within  their  normal  environment.  Access  challenges  because  of  the  existence  of  only  a 
few  experts  could  be  overcome  with  this  method  if  several  novices  were  used  to  create 
different  presentations  and  the  experts  critiqued  the  different  novices.  Another  access 
issue  that  arises  in  some  domains  is  that  experts  are  reluctant  to  participate  due  to 
repercussions,  such  as  erroneously  performing  an  act  while  being  observed  that  could 
have  dire  consequences.  For  critiquing,  as  the  expert  is  in  a  role  other  than  performing 
work,  he  may  see  the  elicitation  as  less  of  a  threat  and  therefore  be  willing  to 
participate.  None  of  the  experts  who  participated  in  this  study  as  critiquers  were 
reluctant  to  participate. 

All  CTA  methods  are  time  consuming,  especially  the  knowledge  elicitation  phase.  One 
original  belief  for  this  study  was  that  by  presenting  the  packaged  representation  of  the 
novice's  process,  the  experts  would  use  less  time  to  comment  than  would  have  been 
used  by  having  the  experienced  analysts  directly  observe  the  novice.  Surprisingly,  the 
novice  took  about  1  hour  and  20  minutes  and  the  experienced  analysts  ranged  from  1 
hour  and  10  minute  to  1  hour  and  21  minutes.  Although  the  times  were  about  the  same, 
the  experienced  analysts  did  not  have  to  sit  through  the  less  informative  times,  such  as 
waiting  for  query  results,  so  the  time  they  spent  in  their  sessions  was  all  productive 
time  focused  on  the  task.  Therefore,  based  on  this  research,  the  critiquing  method  has 
some  increased  efficiencies. 

Repeatability  of  the  method  was  demonstrated  in  this  study  as  the  same  presentation 
was  shown  to  all  six  different  experts  and  all  experts  made  insightful  comments.  A 
total  of  133  relevant  comments  were  made  with  the  range  being  17  to  48.  The 
comments  were  evaluated  to  determine  if  there  was  an  obvious  breakdown  by  year’s 
experience,  prior  knowledge  or  work  experience  and  no  trends  were  found.  This  result 
is  not  surprising  considering  the  small  sample  size.  Repeatability  was  also  indicated  as 
E7  and  ElO  made  similar  comments  when  they  critiqued  the  first  novice  as  when  they 
critiqued  the  second  novice  as  discussed  above.  The  results  of  all  eight  critiques  had 
similarities. 


While  there  is  not  a  direet  relationship  between  the  tasks  investigated  for  intelligenee 
analysis  and  tasks  performed  by  eommanders,  similarity  exists.  Strategy  development 
entails  the  steps  of  mission  analysis,  eourse  of  aetion  development,  eourse  of  aetion 
analysis,  eourse  of  aetion  eomparison  and  eourse  of  aetion  seleetion.  At  a  high  level, 
these  ean  be  eompared  to  the  steps  previously  diseussed  of  information  seleetion, 
eorroboration/resolution  and  story  eonstruetion  as  the  eommander  is  gathering  and 
sorting  through  available  information  on  blue,  red  and  gray  forees,  eomparing  inputs  to 
ensure  his  understanding  is  eorreet  and  then  putting  together  the  appropriate  eourse  of 
aetion  to  address  the  objeetives  based  on  his  understanding  of  the  eomplete  situation. 
The  next  step  in  this  researeh,  then,  would  be  to  eompose  an  hour-long  simulation  of  a 
eommander  in  a  risky,  fast-paeed  situation  as  the  seenario  to  be  eritiqued.  An 
experieneed  eommander,  possibly  one  of  the  Air  Foree’s  mentors,  would  then  be  shown 
the  seenario  and  his  eomments  eaptured.  As  with  this  study,  the  eommander  would  be 
eneouraged  to  eomment  on  what  was  done  well,  what  was  done  poorly  and  what  he 
would  have  done  given  the  same  situation. 
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Problem 


•  In-depth  understanding  of  C2  is  required  for  developing 
accurate,  relevant  training  and  support 


•  Extracting  tacit  knowledge  is  difficult 

-  Few  experts 

-  Unpredictability 

-  Often  joint,  international  positions 


•  Existing  knowledge  elicitation  methods  have  challenges 

-  Grounding  in  context 

-  Accessibility  to  experts  and  relevant  tasks 

-  Laborious  and  time  consuming 

-  Repeatability 
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Gaining  Understanding 


Process  of  Expert  Performing 

Critiquing  Task 


Novice 


K 

Actual/Simulated 

Performing  y 

Task 

Process  of  Novice  Performing  Domain  Task 


CTA 

Investigator 


Watches 


Expert 
as  Evaluator 


Commenting  on  >  Novice  Performance 


Beginning  Note  Sheet 


**EXERCISE  EXERCISE  EXERCISE  EXERCISE** 


In  1996,  the  European  Space  Agency  lost  a  satellite  during  the  first 
qualification  launch  of  a  new  rocket  design.  Give  a  short  briefing  about 
the  basic  facts  of  the  incident:  when  it  was,  why  it  occurred,  and  what  the 
immediate  impacts  were. 


Complete  Novice  Note  Sheets 


^*EXERCISE  EXERCISE  EXERCISE  EXERCISE** 

.  ^  K  -->  rep'ac’‘'^9 

Anane  5  ^  ^ 

Launcher 


4  * 


A  ' 


In  1996,  the  [European  Space  Agency]  lost  a  satellite  during  the  first  qualification  launch  JU* 

of  a  new  rocket  design.  Give  a  short  briefing  about  the  basic  facts  of  the  incident:  'wjjetn  W  .  qQ6) 
it  w^,  why  it  occurred,  and  what  the  immediate  impacts  were.  '  ^ 


over-  I 
confidence 


exclusively. 

did  not  carry 
out  tests  on 


could  wave 
found  problem 


errors  in 

inertial  reference 
system  (IRS, 
backup  failed, too) 

\  Loss  of  guidance 
130  secs  >  L.O. 

Sot  bad  info,  thought 
it  was 

(2.5  miles.  Self  destruct) 


No 

Expected 

Problem 

*-►  Sunstorm 
[-prevent  EM 
storms  on 
sun 


Inquiry  . 

Board  --->  W  (French) 


►  IRS  should  be  turned 

•off  immediately  after  lift  off 
■■  '[test  facility]^  -->  more  thorough  testing  program 

I  "'HT 

!_►  w/as  much  real  equipment  \auncWe 

as  technically  feasible  Next 

•More  transparent  work/w  Ariane  5  partner 
•384$  recovery  from  failure 

- >1 - ^ 

get  well  program  \  /  ^nplanne 


flight 
mid  April 
1997 


what  they  did: 

•rewritten  sw 

-->  in  light  of  lessons 


fill  in 
w/old 
tech 


Made  better  testbed 
analyzed  all  SW 

._>  used  A-^ 

sw 


when 


-5  Jun  1996,  Ariane  5  program,  replacing  Ariane  4  program 

failed, 


why  (back^'P  loSiji^ 

SW  failure -lIRSl  system  malfunction(wrong  system:for  A-4  program 
didnTexpect  problems 
-->  did  not  test 
-->  could  have  prevented 
Impacts-- >  $384  blunder,  lost  year  of  program 

unplanned  flight 

get  well  program 

re-analyze  all  sw  (IRS),  made  more  realistic  sw 


fill  in  Ariane  4 
rocket 


6 


Progressive  Novice  Note  Sheets 


7 


Experts’  Comments  about  Study 


*E5  -  No  prior  knowledge:  I  felt  uncomfortable 

*E6  -  No  prior  knowledge:  I  was  a  little  uncomfortable. 


•E8  -  Partial  prior  knowledge:  I’m  glad  I  was  prepared.  I  was  able  to  draw  my  own 
conclusions. 

*E9  -  Partial  prior  knowledge:  I  would  not  have  been  comfortable  without  at  least 
reading  the  articles,  seeing  what  data  set  he’s  working  with.  If  you  don’t  know  what 
data  set  they  have,  you  can’t  critique,  it’s  not  valid. 

*E7  -  Full  prior  knowledge:  I  was  comfortable  in  the  way  data  was  presented. 

You  might  tell  them  (novices)  but  they  have  to  do  it  themselves.  This  is  good  for 
training  as  they  do  the  work  and  not  just  memorize. 

*E10-  Full  prior  knowledge:  I  had  fun.  I  found  it  intriguing. 


Process  of  Expert  Performing 

Critiquing  Task 


Novice 


K 

Actual/Simulated 

Performing  y 

Task 

Process  of  Novice  Performing  Domain  Task 


CTA 

Investigator 


Watches 


Expert 
as  Evaluator 


Commenting  on  >  Novice  Performance 


Critiquing  by  Expert 
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Discussion 


*  Remember:  Any  method  shapes  the  conditions  of 
observation. 

*  Reiationship  to  C2 

-  Can  be  used  in  conjunction  with  Modeiing  & 
Simuiation 

-  Provides  strong  cue  to  focus 
—  Participatory  role 
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Conclusion 


•  Existing  knowiedge  eiicitation  methods  have 
chaiienges 

-  Grounding  in  context 

-  Accessibility  to  experts  and  relevant  tasks 

-  Laborious  and  time  consuming 

-  Repeatabiiity 

*  Critiquing  used  to  get  an  in-depth  understanding 
of  the  C2  domain  required  for  developing 
accurate,  relevant  training  and  support 
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Backups 
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Generative 


Designing  Toois  for  Peopie 


Abstract 

Patterns  in  Cognitive  Systems  Abstracted  Patterns 


As  Tools  for  Discovery 


Participative 


Design  Seeds:  Reusable 
Concepts  And  Techniques 


Adapted  from  D.  Woods,  2001  14 


Authentic 


Method  Breakdowns 


Families: 

•  Structured  Interview/Elicitation 

•  Performance/Observation 

•  Mapping  Semantic  Space 

•  Functional  Domain  Modeling 

•  Functional  Task  Modeling 

•  Computational  Modeling  Methods 

•  Participatory  Design  Methods 


15 


1,^ 


omparison  of  Elicitation  Methods 


Elicitation  by  Interview 


CTA 

Investigator 


Questions 


Expert 
as  Storyteller 


Telling  About^ 


Past  Cases, 
Experiences 


Elicitation  by  Observation 


CTA 

Investigator 


Watches 


Expert 

as  Practitioner 


Performing 


Actual/Simulated 
Task  _ 


Elicitation  by  Critiquing 


CTA 

Investigator 


Watches 


Expert 
as  Evaluator 


<  I!ommenting  on 


Novice 


Performing 
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Analyst  E7  on  Novice  1 


•Information  Selection: 

-Understand  what  the 
question  really  is 
-Have  a  goal  in  mind 
-Need  to  eheek  intell 
sourees  not  just  open 
eourse 


•Corroborate/Resolution: 

-Be  aware  of  direeted 
sourees 

-Need  multiple  sourees  to 
eonfirm  data 
-Talk  to  other  people 
-Reports  6  months  or  so 
after  an  event  probably 
more  aeeurate 
-Take  open  souree  with  a 
grain  of  salt 

-Human  sourees  have  to 
have  direet  knowledge 


•  Story  Construct: 

-Get  familiar  before 
formulating 

-  Need  to  know  the  audienee 

-  Senior  people  want  pietures, 
graphs 

-  Inelude  basies  if  audienee 
might  not  be  fully  informed 

-  Admit  what  you  don’t  know 
-You  have  to  be  right  with 
what  you  say 

-  Never  eontradiet  yourself 
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Analyst  E7  on  Novice  2 


•Information  Selection: 

-Understand  the  problem 
-Keep  full  task  in  mind 
as  you  read 

-Look  for  patterns  in  the 
title  list 

-Don't  build  results  on 
one  seareh. 


•Corroborate/Resolution: 

-Talk  to  other  analysts  to  see 
to  diseuss  the  problem 
-What  souree  information 
eomes  from  is  very 
important,  loses  validity  if 
2nd  or  3rd  hand  information 
-It's  neeessary  to 
eorroborate  information, 
might  not  use  if  only  in  one 
souree 


•  Story  Construct: 

-Talk  to  other  analysts  to  see  to 
diseuss  the  problem 
-What  souree  information 
eomes  from  is  very  important, 
loses  validity  if  2nd  or  3rd  hand 
information 

-It's  neeessary  to  eorroborate 
information,  might  not  use  if 
only  in  one  souree 
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Analyst  E10  on  Novice  1 


Mnformatio^s3eroon^™ 

-Can  get  wrong  focus  if 
don't  correctly  define 
-Go  back  to  the  person 
asking  question  to  resolve 
ambiguity 

-If  using  on-screen  views 
of  title  list,  make  sure 
window  is  large  enough  to 
notice  patterns  of  large  part 
of  data  set 

-Read  titles  to  get  a  feel  for 
the  data  set 

-Value  of  data  dependent 
upon  source 

-Use  meta  data  in  title  list, 
that  is,  for  example,  the  fact 
one  name  shows  up  a 
multitude  of  times 


•Corroborate/Resolution: 

-Ask  other  analysts  for  opinions 
-Will  have  to  evaluate  data,  not 
just  read  it 

-Language  can  be  considered  a 
high  level  determinant  during 
data  evaluation 
-Watch  for  biased  reporting 
-Be  aware  of  information  that  is 
actually  only  repeating  another 
source 

-Don't  base  assessment  on  only 
a  few  documents 
-Be  aware  complete  details 
might  intentionally  not  be 
revealed 

-Be  aware  information  might  not 
be  directly  stated 


*  Story  Construct: 

-Look  for  patterns 
-Sometimes  setting  down  a 
timeline  helps 
-Use  external  memory,  if 
necessary,  to  jog  memory 
-Diagram  the  info  if  possible 
-Include  implications  in 
assessment 

-Have  list  of  sources  to 
support  assessment 
-Accept  that  time  and 
resource  constraints  affect 
the  possible  result 


19 


Analyst  E10  on  Novice  2 


•Information  Selection: 

-Do  more  than  one  seareh 
-Look  for  implieations 
-Use  information  in  titles 
but  don't  fully  judge 
doeuments  with  that 
-Keep  foeused  on  task 
-Look  for  information  over 
time 


•Corroborate/Resolution: 

-Compare  information  over 
time  to  look  for  ehanges. 
-Always  have  to  evaluate 
sourees. 

-Use  several  sourees 
-Look  for  independent 
reviews 


•  Story  Construct: 

-Be  prepared  for  in-depth 
questions 

-Inelude  implieations. 
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